Visual tracking method by color information

ABSTRACT

The present invention relates to a method of visual tracking with use of color. The visual tracking method according to the present invention comprise the step of 3-dimensional color modeling of the target, the step of target recognition for initially perceiving the target, and the step of visual tracking for repetitive visual tracking.  
     In order to be robust to irregular or abrupt changes of illumination, the present invention provides a visual tracking method in which characteristics of the photographing element (CCD or CMOS) used by the camera is analyzed beforehand and the results of the analysis is modeled by B-spline curve, then allowing effects of the real-time visual tracking and the adaptability to situations of rapid target movement.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to a method of visual tracking withuse of color. More particularly, the invention relates to visual targettracking method applicable for various fields in such techniques ascorrecting important parts of recorded images for broadcasting stations,tracking human face in unmanned monitoring systems, tracking speakersfor remote conference or remote conversation trough communicationnetworks, setting face-contour for face recognition in security systems,tracking of a specified target, and so forth.

[0003] The need for tracking or recognizing a target having a specificcolor is increased in various application fields like broadcasting,unmanned monitoring systems, security systems, remote conference troughcommunication networks, control of unmanned flying object, unmanneddocking systems, etc. Considering the fact that majority of informationhuman beings obtain is through visual means, the visual trackingtechniques is expected to further expand their application fields in thefuture.

[0004] 2. Description of the Related Art

[0005] In conventional visual tracking techniques using colorinformation, colors of the target have been represented by color models,which are known to be robust to illumination changes such as innormalized Red-Green-Blue (R-G-B) space or Hue-Saturation-Intensity(H-S-I) space.

[0006] However, these color models well accommodate uniform changes ofillumination, but show limitations in the cases where the brightness ofthe target changes irregularly due to angle changes between light sourceand target or the brightness changes abruptly.

[0007] For example, when a man is walking through a gallery whoseceiling is equipped with fluorescent lights at uniform intervals, therelative position of the immediately affecting fluorescent light to theman keeps changing and, therefore, the brightness of the man's facekeeps changing. And depending on the direction the man is moving, itoften probable that one side of the face gets darken while the otherside gets lighted. Moreover, surface reflectiveness of human face ishardly uniform due to secretion like sweat and it is extremely unlikelyfor the whole face to have uniform illumination changes.

[0008] In other words, since color distribution changes as illuminationintensity varies, the conventional visual tracking techniques haveproblems in using a color model normalized and set-up for uniformillumination changes.

SUMMARY OF THE INVENTION

[0009] It is an object of the present invention to resolve theaforementioned problems by providing a method of visual tracking withuse of color information which is robust to irregular or abrupt changesof illumination.

[0010] It is another object of the present invention to provide a visualtracking method using color information in which characteristic of thephotographing element of the camera is analyzed and the results of theanalysis is modeled by B-spline curve, then allowing real-time visualtracking and application for the situations of rapid movement of thetarget.

[0011] The present invention proposes a visual tracking method in whichthe photographing element of camera (CCD or CMOS) is analyzed for itscharacteristics to the brightness and the analysis results are modeledbeforehand so that it resolves the problems of the conventional visualtracking algorithm which is not adaptable to non-uniform or irregularchange of illumination.

[0012] The present invention also proposes a motion accelerationpredictor for improving the speed of visual tracking.

[0013] The present invention provides a visual tracking method usingcolor information comprising: three-dimensional color modeling step inwhich images obtained under various illumination conditions are analyzedand thereby the photographing characteristics of the camera as to thetarget is represented by a three-dimensional model; target recognitionstep in which judgement is made by the difference between the previousand the current images about whether or not a new target object appears,target region is located by applying the color model in saidthree-dimensional color modeling step, and the final decision is made asto whether or not visual tracking is to be performed for the targetdepending on the shape analysis of the target region and the third stepof visual tracking in which an arbitrary pixel is monitored and judgedusing said color model if it belongs to the target region and at thesame time the judgement process has to be adaptable to the movementspeed of the target by estimating the movement of the target region.

[0014] The above and other features and advantages of the presentinvention will be more clearly understood for those skilled in the artfrom the following detailed description taken in conjunction with theaccompanying drawings, which form a part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015]FIG. 1 is a schematic block diagram representing the concept ofthe visual tracking method according to the present invention;

[0016]FIG. 2 is a graph showing the average hue to the brightness valuemodeled by the present invention;

[0017]FIG. 3 is a graph showing the standard deviation of hue to thebrightness value modeled by the present invention;

[0018]FIG. 4 is a graph showing the average saturation to the brightnessvalue modeled by the present invention; and

[0019]FIG. 5 is a graph showing the standard deviation of saturation tothe brightness value modeled by the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENT

[0020] Hereinafter, the present invention is described in detail byreferring to the accompanying drawings.

[0021] First of all, the visual tracking method according to the presentinvention comprise the step of three-dimensional color modeling of thetarget (S100), the step of target recognition for initially perceivingthe target (S200), and the step of visual tracking for repetitive visualtracking (S300); and the basic concept of the invention is schematicallyshown in FIG. 1.

[0022] The first step of three-dimensional color modeling (S100) is thestep for establishing a three-dimensional color model and has to beperformed before initiation of the visual tracking.

[0023] The above color model consists of four quadratic functionsrepresenting average and standard deviation of target hue with respectto brightness changes and average and standard deviation of targetsaturation. And these four quadratic functions constitute a singleGaussian function.

[0024] For this purpose, the model is obtained by dividing the wholeregion subject to brightness changes of the image into n regions withregular sizes and by approximating two adjacent regions using quadraticcurves, accomplishing the model for the whole region.

[0025] Here, each region is represented by a quadratic function such asEqn. 1 along with boundary conditions of the quadratic curvesrepresented by Eqns. 2-4. In other words, the model is built by theregion curve (Eqn. 1) along with boundary value condition (Eqn. 2),continuity condition at the boundary (Eqn. 3), and second derivativecondition for the region curve (Eqn. 4) as shown in the followings.

f _(i)(x)=a _(i) ·x ² +b _(i) ·x+c _(i).  [Eqn. 1]

f _(i)(x _(i+1))=a _(i) ·x _(i) ² +b _(i) ·x _(i) +c _(i) =f(x _(i)),for i=1, 2, . . . n−1.

f _(i)(x _(i+1))=a _(i) ·x _(i+1) ² +b _(i) ·x _(i+1) +c _(i) =f(x_(i+1)), for i=1,2 . . . n−1.  [Eqn. 2]

2·a _(i) ·x _(i+1) +b ₁=2·a _(i+1) +b _(i+1), for i=1,2, . . .n−1.  [Eqn. 3]

a ₁=0.  [Eqn. 4]

[0026] Herein, x is the brightness of the concerning pixel, and f(x) isthe average or standard deviation of the hue or the average or standarddeviation of the saturation for the given brightness x.

[0027] If the whole range of brightness is divided into 5, therelational equation for obtaining the coefficients of the quadraticcurve for each region can be represented by a single matrix equation ofEqn. 5 as shown in the followings. $\begin{matrix}{\left( {b_{1}c_{1}a_{2}b_{2}c_{2}a_{3}b_{3}c_{3}a_{4}b_{4}c_{4}} \right)^{T} = {\left\lbrack \quad \begin{matrix}x_{1} & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & x_{2}^{2} & x_{2} & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & x_{3}^{2} & x_{3} & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & x_{4}^{2} & x_{4} & 1 \\x_{2} & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & x_{3}^{2} & x_{3} & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & x_{4}^{2} & x_{4} & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & x_{5}^{2} & x_{5} & 1 \\1 & 0 & {{- 2}x_{2}} & {- 1} & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & {2x_{3}} & 1 & 0 & {{- 2}x_{3}} & {- 1} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & {2x_{4}} & 1 & 0 & {{- 2}x_{4}} & {- 1} & 0\end{matrix}\quad \right\rbrack^{- 1}\begin{bmatrix}{f\left( x_{1} \right)} \\{f\left( x_{2} \right)} \\{f\left( x_{3} \right)} \\{f\left( x_{4} \right)} \\{f\left( x_{2} \right)} \\{f\left( x_{3} \right)} \\{f\left( x_{4} \right)} \\{f\left( x_{5} \right)} \\0 \\0 \\0\end{bmatrix}}} & \left\lbrack {{Eqn}.\quad 5} \right\rbrack\end{matrix}$

[0028] Herein, (a_(i), b_(i), c_(i)) for i=1,2,3, and 4 denotecoefficients for quadratic curve representing each region.

[0029] Based on four relational equations obtained above,three-dimensional color model is represented by the following Eqns. 6and 7; i.e.,

H _(m)(i)−T _(h) ·H _(σ)(i)≦H _(3D)(i)≦H _(m)(i)+T _(h) ·H_(σ)(i)  [Eqn. 6]

S _(m)(i)−T _(h) ·S _(σ)(i)≦S _(3D)(i)≦S _(m)(i)+T _(h) ·S_(σ)(i)  [Eqn. 7]

[0030] Herein, H_(m)(i) for 0≦i≦255 is the function of hue average,H_(σ)(i) for 0≦i≦255 is the function of hue standard deviation, S_(m)(i)for 0<i≦255 is the function of saturation average, S₉₄(i) for 0≦i≦255 isthe function of saturation standard deviation, and H_(3D)(i) for 0≦i≦255and S_(3D)(i) for 0≦i≦255 are the color model functions.

[0031] In the second step of target recognition (S200), judgement ismade about whether a new target object appears by the difference betweenthe previous and the current images, target region is located byapplying the color model newly proposed in the previous step, and thefinal decision is made as to whether or not visual tracking is to beperformed for the target by shape analysis of the target region.

[0032] The aforementioned shape analysis is performed by comparing theactual shape of the image in the target region against the outlinedshape inputted beforehand. For example, if human face is tracked, eggshape is chosen as the reference for contour comparison; if a ball istracked, circular shape is chosen as comparison reference.

[0033] In the third step of visual tracking (S300) , the recognizedtarget region is continuously tracked.

[0034] In this step, an arbitrary pixel is monitored and judgedcontinuously if it belongs the target region using the formulated colormodel, while target region movement is estimated and the judgementprocess has to be accommodate the movement speed of the target.

[0035] Kalman filter technique might be considered for the movementprediction. However, it has a slow algorithmic speed due to its heavyarithmetic load and shows drawbacks that prompt adaptation is impossiblefor abrupt movement because it utilizes too much of the pastinformation.

[0036] To cope with this difficulty, the present invention employs asimple movement tracking method, which uses analysis results of the pastthree images as shown in Eqn. 8; i.e., $\begin{matrix}{{P_{m}(i)} = {\frac{\left( \frac{F_{m}\left( {i - 1} \right)}{t} \right)}{t} \approx \frac{{F_{m}\left( {i - 1} \right)} - {2 \cdot {F_{m}\left( {i - 2} \right)}} + {F_{m}\left( {i - 3} \right)}}{\Delta \quad {t \cdot \Delta}\quad t}}} & \left\lbrack {{Eqn}.\quad 8} \right\rbrack\end{matrix}$

[0037] Herein, P_(m)(i) is the predicted acceleration of the targetregion in the i-th image, F_(m)(i) is the position of the target regionin the i-th image, and Δt is the time increment between the i-th and(i−1)-th images.

[0038]FIGS. 2 through 5 show four curves which are modeled for facetracking according to the present invention under various illuminationconditions.

[0039] In FIGS. 2 through 5, the abscissa represents brightness valuewhile the ordinate represents average and standard deviation of hue andthose of saturation, respectively.

[0040] According to the present invention, the brightness of the facechanges depending on the relative position of the ceiling fluorescentlamp to the face, and the tracking works continuously even ifillumination is abruptly darkened.

[0041] As shown previously, the present invention allows adaptableperformance of camera when a target with a specific color is trackedunder irregular or non-uniform change of illumination. In other words,the camera photographing characteristics as to the target arethree-dimensionally modeled by analyzing the images obtained undervarious brightness conditions, thus enhancing reliability againstillumination changes, and a simple acceleration predictor with a lightarithmetic burden is employed for adaptation for movement speed changeof the target. So that the present invention has the effect which can beutilized in various application fields like broadcasting, unmannedmonitoring systems, security systems, remote conference troughcommunication networks, control of unmanned flying object, unmanneddocking systems, and so forth.

[0042] Although the present invention has been described and illustratedin connection with the specific embodiments, it will be apparent forthose skilled in the art that various modifications and changes may bemade without departing from the idea and scope of the present inventionset forth in the appended claims.

What is claimed is:
 1. A visual tracking method using color informationcomprising: three-dimensional color modeling step (S100) in which imagesobtained under various illumination conditions are analyzed and therebythe photographing characteristics of the camera as to the target isrepresented by a three-dimensional model; target recognition step (S200)in which judgement is made by the difference between the previous andthe current images about whether or not a new target object appears,target region is located by applying the color model in saidthree-dimensional color modeling step (S100), and the final decision ismade as to whether or not visual tracking is to be performed for thetarget depending on the shape analysis of the target region; and thethird step of visual tracking (S300) in which an arbitrary pixel ismonitored and judged using said color model if it belongs to the targetregion and at the same time the judgement process has to be adaptable tothe movement speed of the target by estimating the movement of thetarget region.
 2. A visual tracking method using color informationcomprising: three-dimensional color modeling step (S100) in which thecolor model is constituted by four quadratic functions representingaverage and standard deviation of target hue and average and standarddeviation of target saturation, and said color model for the wholeregion is accomplished by dividing the whole region subject to imagebrightness changes into n regions with regular sizes and byapproximating each region using a quadratic curve and making the fourquadratic functions constitute a single Gaussian function; targetrecognition step (S200) in which judgement is made by the differencebetween the previous and the current images about whether or not a newtarget object appears, target region is located by applying the colormodel in said three-dimensional color modeling step (S100), and thefinal decision is made as to whether or not visual tracking is to beperformed for the target depending on the shape analysis of the targetregion; and the third step of visual tracking (S300) in which anarbitrary pixel is monitored and judged using said color model if itbelongs to the target region and at the same time the judgement processhas to be adaptable to the movement speed of the target by estimatingthe movement of the target region.
 3. A visual tracking method usingcolor information of claim 2 wherein said three-dimensional color modelin three-dimensional color modeling step (S100) is formulated byequational relationship of H _(m)(i)−T _(h) ·H _(σ)(i)≦H _(3D)(i)≦H_(m)(i)+T _(h) ·H _(σ)(i) and S _(m)(i)−T _(h) ·S _(σ)(i)≦S _(3D)(i)≦S_(m)(i)+T _(h) ·S _(σ)(i), where H_(m)(i) for 0≦i≦255 is the function ofhue average, H_(σ)(i) for 0≦i≦255 is the function of hue standarddeviation, Sm(i) for 0≦i≦255 is the function of saturation average,S_(σ)(i) for 0≦i≦255 is the function of saturation standard deviation,and H_(3D)(i) for 0≦i≦255 and S_(3D)(i) for 0≦i≦255 are the color modelfunctions.
 4. A visual tracking method using color information of claim2 wherein the predicted acceleration in said visual tracking step (S300)is given by the equation of${{P_{m}(i)} = {\frac{\left( \frac{F_{m}\left( {i - 1} \right)}{t} \right)}{t} \approx \frac{{F_{m}\left( {i - 1} \right)} - {2 \cdot {F_{m}\left( {i - 2} \right)}} + {F_{m}\left( {i - 3} \right)}}{\Delta \quad {t \cdot \Delta}\quad t}}},$

where P_(m)(i) is the predicted acceleration of the target region in thei-th image, F_(m)(i) is the position of the target region in the i-thimage, and Δt is the time increment between the i-th and (i−1)-thimages.